Skip to content

Add ScanMalware expansion module - #803

Merged
adulau merged 1 commit into
MISP:mainfrom
jonaslejon:feat/scanmalware-expansion
Aug 29, 2026
Merged

Add ScanMalware expansion module#803
adulau merged 1 commit into
MISP:mainfrom
jonaslejon:feat/scanmalware-expansion

Conversation

@jonaslejon

Copy link
Copy Markdown
Contributor

New expansion + hover module: ScanMalware

Adds misp_modules/modules/expansion/scanmalware.py, enriching domain, hostname, url,
ip-src, ip-dst and domain|ip attributes from the public
ScanMalware API, returning misp_standard results built with PyMISP.

No API key. The API is anonymous and a key only raises the rate limit
(600 req/min anonymous), so moduleconfig is ["api_url", "max_results"] and
requirements is empty. The module works on a fresh install with nothing configured.

Behaviour

  • A domain or hostname is queried against two different populations, not one.
    /api/v1/ct/dns/ returns subdomains seen in Certificate Transparency DNS records;
    /api/v1/hosts/ returns hosts a browser actually resolved and requested while rendering
    pages on that domain. The second is what certificate-based sources structurally cannot
    see: dev and staging hosts that hold no certificate. Querying only the first loses them.
  • A URL returns the most recent completed sandbox scan of its host, with the verdict, a
    link to the full report, and the IPs contacted while rendering.
  • An IP returns the URLs in the public archive that resolved to that address.
  • Results are capped, and the cap says so. max_results defaults to 200. Unbounded,
    cloudflare.com returns 1,207 attributes, which makes an event unusable. The cap and
    any truncation the API itself reports are both added as visible text attributes rather
    than applied silently, so an analyst can tell a shortened list from a complete one.
  • Only the queried indicator leaves the MISP instance. Requests carry a
    scanmalware-misp/1.0 User-Agent.

Measured output

domain  hkr.se            ->  70 attributes  (domain 70)
domain  cloudflare.com    -> 201 attributes  (domain 200 + the cap notice)
url     https://wpsec.com/->  82 attributes  (verdict text, report link, 80 contacted IPs)
ip-src  172.67.216.2      ->   3 attributes  (archive summary + resolved hosts)

Tests

tests/test_scanmalware.py, 12 tests, no network access. They cover the cases that would
otherwise degrade quietly: wildcard certificate entries (*.example.com) being dropped
rather than emitted as hosts, truncation and the result cap both being reported, a URL
being reduced to its host before lookup, an unreachable API degrading instead of raising,
and no auth header ever being sent.

Clean under the repository's own black (line length 120, preview,
string_processing), isort and flake8 settings.

Note on documentation: I deliberately have not committed regenerated
documentation/ output. Running documentation/generate.py locally drops every module
whose optional dependencies are missing, which produced a 4,600-line deletion across
unrelated modules. Happy to add the generated entries if you would like them in this PR,
but it seemed safer to leave that to a run with the full dependency set. The logo is
included since moduleinfo references it.


Disclosure: I run ScanMalware, so this is a vendor-submitted module. Flagging it up
front rather than leaving it to be found in review.

Queries the public ScanMalware API for a domain, hostname, URL or IP attribute.
No API key is required: the API is anonymous, and a key only raises the rate
limit.

For a domain or hostname it returns two different populations, subdomains seen in
Certificate Transparency DNS records and hosts a browser actually resolved and
requested while rendering pages on that domain. Dev and staging hosts that hold
no certificate appear only in the second.

For a URL it returns the most recent sandboxed scan of that host with its verdict
and a link to the report, and for an IP the URLs in the public archive that
resolved to that address.

Results are capped at max_results attributes, 200 by default, because a busy
domain can return well over a thousand subdomains. The cap and any truncation
reported by the API are both surfaced as attributes rather than applied silently.
@adulau

adulau commented Aug 29, 2026

Copy link
Copy Markdown
Member

Thanks a lot for the module.

We did some tests with some non-existing random hash values and it gives a value. Not sure if this is expected. Not really related to the module but more the service.

@adulau
adulau merged commit 32dd244 into MISP:main Aug 29, 2026
7 checks passed
@jonaslejon

Copy link
Copy Markdown
Contributor Author

Thanks for merging it, and for testing with random values.

You were right that it was the service. The API was accepting any string as a domain and
answering 200 with a fully-formed object that echoed the input back, so a random hash
produced an answer-shaped response with zero counters. That has been fixed: those
endpoints now return 422 with a validation message, and valid but awkward inputs
(punycode, IDN, trailing dot, deep subdomains) still work.

curl -s "https://scanmalware.com/api/v1/ct/dns/$(openssl rand -hex 16)"
{"error":"Validation failed","details":["... must be a hostname with at least two labels ..."]}

The module needed a change too: it was turning that rejection into "No ScanMalware
results for this attribute", which is the same misleading answer in the other direction.
Chasing it also turned up a real bug in what I shipped: for a URL attribute the module
reported the newest scan of the host, so https://github.io/ came back with the
verdict for an unrelated subdomain. Both fixed in #806.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants